Development and evaluation of a text analytics algorithm for automated application of national COVID-19 shielding criteria in rheumatology patients.

INTRODUCTION: At the beginning of the COVID-19 pandemic, the UK's Scientific Committee issued extreme social distancing measures, termed 'shielding', aimed at a subpopulation deemed extremely clinically vulnerable to infection. National guidance for risk stratification was based on patients' age, comorbidities and immunosuppressive therapies, including biologics that are not captured in primary care records. This process required considerable clinician time to manually review outpatient letters. Our aim was to develop and evaluate an automated shielding algorithm by text-mining outpatient letter diagnoses and medications, reducing the need for future manual review. METHODS: Rheumatology outpatient letters from a large UK foundation trust were retrieved. Free-text diagnoses were processed using Intelligent Medical Objects software (Concept Tagger), which used interface terminology for each condition mapped to Systematized Medical Nomenclature for Medicine-Clinical Terminology (SNOMED-CT) codes. We developed the Medication Concept Recognition tool (Named Entity Recognition) to retrieve medications' type, dose, duration and status (active/past) at the time of the letter. Age, diagnosis and medication variables were then combined to calculate a shielding score based on the most recent letter. The algorithm's performance was evaluated using clinical review as the gold standard. The time taken to deploy the developed algorithm on a larger patient subset was measured. RESULTS: In total, 5942 free-text diagnoses were extracted and mapped to SNOMED-CT, with 13 665 free-text medications (n=803 patients). The automated algorithm demonstrated a sensitivity of 80% (95% CI: 75%, 85%) and specificity of 92% (95% CI: 90%, 94%). Positive likelihood ratio was 10 (95% CI: 8, 14), negative likelihood ratio was 0.21 (95% CI: 0.16, 0.28) and F1 score was 0.81. Evaluation of mismatches revealed that the algorithm performed correctly against the gold standard in most cases. The developed algorithm was then deployed on records from an additional 15 865 patients, which took 18 hours for data extraction and 1 hour to deploy. DISCUSSION: An automated algorithm for risk stratification has several advantages including reducing clinician time for manual review to allow more time for direct care, improving efficiency and increasing transparency in individual patient communication. It has the potential to be adapted for future public health initiatives that require prompt automated review of hospital outpatient letters.

Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study.

OBJECTIVE: Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics. MATERIALS AND METHODS: Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics. RESULTS: On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5. CONCLUSIONS: Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

Evaluation of a Concept Mapping Task Using Named Entity Recognition and Normalization in Unstructured Clinical Text.

In this pilot study, we explore the feasibility and accuracy of using a query in a commercial natural language processing engine in a named entity recognition and normalization task to extract a wide spectrum of clinical concepts from free text clinical letters. Editorial guidance developed by two independent clinicians was used to annotate sixty anonymized clinic letters to create the gold standard. Concepts were categorized by semantic type, and labels were applied to indicate contextual attributes such as negation. The natural language processing (NLP) engine was Linguamatics I2E version 5.3.1, equipped with an algorithm for contextualizing words and phrases and an ontology of terms from Intelligent Medical Objects to which those tokens were mapped. Performance of the engine was assessed on a training set of the documents using precision, recall, and the F1 score, with subset analysis for semantic type, accurate negation, exact versus partial conceptual matching, and discontinuous text. The engine underwent tuning, and the final performance was determined for a test set. The test set showed an F1 score of 0.81 and 0.84 using strict and relaxed criteria respectively when appropriate negation was not required and 0.75 and 0.77 when it was. F1 scores were higher when concepts were derived from continuous text only. This pilot study showed that a commercially available NLP engine delivered good overall results for identifying a wide spectrum of structured clinical concepts. Such a system holds promise for extracting concepts from free text to populate problem lists or for data mining projects.

Integration of Postcoordination Content into a Clinical Interface Terminology to Support Administrative Coding.

BACKGROUND: Clinical interface terminologies (CITs) consist of terms designed for clinical documentation and, through mappings to standardized vocabularies, to support secondary uses of patient data, including clinical decision support, quality measurement, and billing for health care services. The latter purpose requires maps to administrative coding systems, such as the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM), for diagnoses in the United States. OBJECTIVES: The transition from ICD-9-CM to ICD-10-CM posed a challenge to CIT users due to the substantially increased details in ICD-10-CM. To address this, we developed a content layer within a CIT that provides postcoordination prompts for the details required for accurate ICD-10-CM coding. METHODS: We developed content to support prompting for and capture of additional information specified by the user in a single, clinically relevant term that is added to the patient's record, and whose mapping to other coding systems (like Systematized Nomenclature of Medicine-Clinical Terms [SNOMED CT]) reflects the details added during postcoordination. We worked with clinical information system developers to incorporate this into user interfaces, and with end-users to refine the design. RESULTS: While the prompts were designed around the precoordinated elements implicit in ICD-10-CM, irregularities in ICD-10-CM required some additional design measures, such as providing postcoordination options that interpolate gaps in ICD-10-CM to avoid user confusion. The system we describe has been implemented by ∼30,000 health care provider organizations, with content that covers the vast majority of encounter diagnoses. User feedback has been largely positive, though concerns have been raised about expanding postcoordination content beyond that required for ICD-10-CM coding. CONCLUSION: We have demonstrated the design and development of what, to our knowledge, is the first system that uses postcoordination to capture ICD-10-CM-relevant details in a CIT while also reflecting the details added by the user in maps to other vocabularies.

Effectiveness of SMS Technology on Timely Community Health Worker Follow-Up for Childhood Malnutrition: A Retrospective Cohort Study in sub-Saharan Africa.

BACKGROUND: The Millennium Villages Project facilitated technology-based health interventions in rural under-resourced areas of sub-Saharan Africa. Our study examined whether data entry using SMS compared with paper forms by community health workers (CHWs) led to higher proportion of timely follow-up visits for malnutrition screening in under-5 children in Ghana, Rwanda, Senegal, and Uganda. METHODS: Children under 5 years were screened for malnutrition every 90 days by CHWs using mid-upper arm circumference (MUAC) readings. CHWs used either SMS texts or paper forms to enter MUAC data. Reminder texts were sent at 15 days before follow-up was needed. Chi-square tests assessed proportion of timely follow-up visits within 90 days between SMS and paper groups. Logistic regression analysis was conducted in a step-wise multivariate model. Post-hoc power calculations were conducted to verify strength of associations. RESULTS: SMS data entry was associated with a higher proportion of timely malnutrition follow-up visits compared with paper forms across all sites. The association was strongest with consistent SMS use over consecutive visits. SMS use at the first of 2 consecutive visits was most effective, highlighting the importance of SMS reminder alerts. CONCLUSIONS: SMS technology with reminders increased timely CHW malnutrition screening visits for under-5 children in Ghana, Rwanda, Senegal, and Uganda, highlighting the importance of such technology for improving health worker behavior in low-resource settings.

Validation of the Behavior of a Knowledge Base Implementing Clinical Guidelines for Point-of-Care Antiretroviral Toxicity Monitoring.

This study investigated the automated detection of antiretroviral toxicities in structured electronic health records data. The evaluation compared responses generated by 5 clinical pharmacists and 1 prototype knowledge-based application for 15 randomly selected test cases. The main outcomes were inter-subject dissimilarity of responses quantified by the Jaccard distance, and the mean proportion of correct responses by each subject. The statistical differences in inter-subject Jaccard distances suggested that the prototype was inferior to clinical pharmacists in the detection of possible antiretroviral toxicity associations from structured data. The reason for dissimilarities was attributable to inadequate domain coverage by the prototype. The differences in the mean proportion of correct responses between the clinical pharmacists and the prototype were statistically indistinguishable. Overall, this study suggests that knowledge-based applications have the potential to support automated detection of antiretroviral toxicities from structured patient records. Furthermore, the study demonstrates a systematic approach for validating such applications quantitatively.

Development and Deployment of the OpenMRS-Ebola Electronic Health Record System for an Ebola Treatment Center in Sierra Leone.

BACKGROUND: Stringent infection control requirements at Ebola treatment centers (ETCs), which are specialized facilities for isolating and treating Ebola patients, create substantial challenges for recording and reviewing patient information. During the 2014-2016 West African Ebola epidemic, paper-based data collection systems at ETCs compromised the quality, quantity, and confidentiality of patient data. Electronic health record (EHR) systems have the potential to address such problems, with benefits for patient care, surveillance, and research. However, no suitable software was available for deployment when large-scale ETCs opened as the epidemic escalated in 2014. OBJECTIVE: We present our work on rapidly developing and deploying OpenMRS-Ebola, an EHR system for the Kerry Town ETC in Sierra Leone. We describe our experience, lessons learned, and recommendations for future health emergencies. METHODS: We used the OpenMRS platform and Agile software development approaches to build OpenMRS-Ebola. Key features of our work included daily communications between the development team and ground-based operations team, iterative processes, and phased development and implementation. We made design decisions based on the restrictions of the ETC environment and regular user feedback. To evaluate the system, we conducted predeployment user questionnaires and compared the EHR records with duplicate paper records. RESULTS: We successfully built OpenMRS-Ebola, a modular stand-alone EHR system with a tablet-based application for infectious patient wards and a desktop-based application for noninfectious areas. OpenMRS-Ebola supports patient tracking (registration, bed allocation, and discharge); recording of vital signs and symptoms; medication and intravenous fluid ordering and monitoring; laboratory results; clinician notes; and data export. It displays relevant patient information to clinicians in infectious and noninfectious zones. We implemented phase 1 (patient tracking; drug ordering and monitoring) after 2.5 months of full-time development. OpenMRS-Ebola was used for 112 patient registrations, 569 prescription orders, and 971 medication administration recordings. We were unable to fully implement phases 2 and 3 as the ETC closed because of a decrease in new Ebola cases. The phase 1 evaluation suggested that OpenMRS-Ebola worked well in the context of the rollout, and the user feedback was positive. CONCLUSIONS: To our knowledge, OpenMRS-Ebola is the most comprehensive adaptable clinical EHR built for a low-resource setting health emergency. It is designed to address the main challenges of data collection in highly infectious environments that require robust infection prevention and control measures and it is interoperable with other electronic health systems. Although we built and deployed OpenMRS-Ebola more rapidly than typical software, our work highlights the challenges of having to develop an appropriate system during an emergency rather than being able to rapidly adapt an existing one. Lessons learned from this and previous emergencies should be used to ensure that a set of well-designed, easy-to-use, pretested health software is ready for quick deployment in future.

Using Natural Language Processing and Network Analysis to Develop a Conceptual Framework for Medication Therapy Management Research.

This paper describes a theory derivation process used to develop a conceptual framework for medication therapy management (MTM) research. The MTM service model and chronic care model were selected as parent theories. Review article abstracts targeting medication therapy management in chronic disease care were retrieved from Ovid Medline (2000-2016). Unique concepts in each abstract were extracted using MetaMap and their pairwise cooccurrence determined. The information was used to construct a network graph of concept co-occurrence that was analyzed to identify content for the new conceptual model. 142 abstracts were analyzed. Medication adherence is the most studied drug therapy problem and co-occurred with concepts related to patient-centered interventions targeting self-management. The enhanced model consists of 65 concepts clustered into 14 constructs. The framework requires additional refinement and evaluation to determine its relevance and applicability across a broad audience including underserved settings.

Modelling the Medication Management System for Resource Limited Settings: A Formal Representation of the Prescribing and Dispensing Phases.

We propose a conceptual data model for relational databases targeting the prescribing and dispensing phases of the medication management system. The model was developed using recommendations from existing standards and guidelines, with necessary modifications made to suit adoption in resource-limited settings. We present the model as an entity-relationship diagram with 10 entities, 12 relationships and 48 attributes. It is our hope that this work will help mitigate barriers in the implementation of electronic prescribing and dispensing standards in the developing world.

Design and development of an EMR for Ebola Treatment Centers in Sierra Leone using OpenMRS.

Ebola treatment presents unique challenges for medical records because strict infection control requirements rule out most conventional record-keeping systems. We used the OpenMRS platform to rapidly develop an EMR system for the recently opened Kerry Town, Sierra Leone Ebola Treatment Centre. This system addresses the need for recording patient data and communicating it between the infectious and non-infectious zones, and is specifically designed for maximum usability by staff wearing cumbersome protective equipment. This platform is interoperable with other key eHealth systems in the country, and is extensible to other sites and diseases.

Towards a Tool for Malaria Supply Chain Management Improvement in Rural Ghana.

The maintenance of adequate quantities of antimalarial medicines and rapid diagnostic tests (RDTs) at health facilities in rural areas of sub-Saharan Africa is a challenging task because of poor supply chain management. Antimalarial stock-outs in the communities could lead patients (that need to travel long distances to get medications) to remain untreated, develop severe malaria and die. A prototype to improve the management of health commodities in rural Ghana through the visualization of current stock levels and the forecasting of commodities is proposed.

Towards a Clinical Decision Support System for Drug Allergy Management: Are Existing Drug Reference Terminologies Sufficient for Identifying Substitutes and Cross-Reactants?

Drug allergy cross-reactivity checking is an important component of electronic health record systems. Currently, a single, open-source medication dictionary that can provide this function does not exist. In this study, we assessed the feasibility of using RxNorm and NDF-RT (National Drug File--Reference Terminology) for allergy management decision support. We evaluated the performance of using the Pharmacological Class, Mechanism of Action and Chemical Structure NDF-RT classifications in discriminating between safe and cross-reactive alternatives to a sample of common drug allergens. The positive predictive values for the three approaches were 96.3%, 99.3% and 96.2% respectively. The negative predictive values were 94.7%, 56.8% and 92.6%. Our findings suggest that in the absence of an established medication allergy classification system, using the Pharmacologic Class and Chemical Structure classifications in NDF-RT may still be effective for discriminating between safe and cross-reactive alternatives to potential allergens.

Health worker perceptions of integrating mobile phones into community case management of malaria in Saraya, Senegal.

BACKGROUND: Although community case management of malaria increases access to life-saving care in isolated settings, it contends with many logistical challenges. Mobile phone health information technology may present an opportunity to address a number of these barriers. METHODS: Using the wireless adaptation of the technology acceptance model, this study assessed availability, ease of use, usefulness, and job relevance of mobile phones by health workers in Saraya, Senegal. RESULTS: This study conducted seven key informant interviews with government health workers, and three focus groups and 76 surveys with lay health workers. Principal findings included that mobile phones are already widely available and used, and that participants valued using phones to address training, stock management, programme reporting, and transportation challenges. CONCLUSIONS: By documenting widespread use of mobile phones and health worker perceptions of their most useful applications, this paper provides a framework for their integration into the community case management of malaria programme in Saraya, Senegal.

A usability study of a mobile health application for rural Ghanaian midwives.

INTRODUCTION: Midwives in rural Ghana work at the frontline of the health care system, where they have access to essential data about the patient population. However, current methods of data capture, primarily pen and paper, make the data neither accessible nor usable for monitoring patient care or program evaluation. Electronic health (eHealth) systems present a potential mechanism for enhancing the roles of midwives by providing tools for collecting, exchanging, and viewing patient data as well as offering midwives the possibility for receiving information and decision support. Introducing such technology in low-resource settings has been challenging because of low levels of user acceptance, software design that does not match the end-user environment, and/or unforeseen challenges such as irregular power availability. These challenges are often attributable to a lack of understanding by the software developers of the end users' needs and work environment. METHODS: A mobile health (mHealth) application known as mClinic was designed to support midwife access to the Millennium Village-Global Network, an eHealth delivery platform that captures data for managing patient care as well as program evaluation and monitoring, decision making, and management. We conducted a descriptive usability study composed of 3 phases to evaluate an mClinic prototype: 1) hybrid lab-live software evaluation of mClinic to identify usability issues; 2) completion of a usability questionnaire; and 3) interviews that included low-fidelity prototyping of new functionality proposed by midwives. RESULTS: The heuristic evaluation identified usability problems related to 4 of 8 usability categories. Analysis of usability questionnaire data indicated that the midwives perceived mClinic as useful but were more neutral about the ease of use. Analysis of midwives' reactions to low-fidelity prototypes during the interview process supported the applicability of mClinic to midwives' work and identified the need for additional functionality. DISCUSSION: User acceptance is essential for the success of any mHealth implementation. Usability testing identified mClinic development flaws and needed software enhancements.

Mobile phones and social structures: an exploration of a closed user group in rural Ghana.

BACKGROUND: In the Millennium Villages Project site of Bonsaaso, Ghana, the Health Team is using a mobile phone closed user group to place calls amongst one another at no cost. METHODS: In order to determine the utilization and acceptability of the closed user group amongst users, social network analysis and qualitative methods were used. Key informants were identified and interviewed. The key informants also kept prospective call journals. Billing statements and de-identified call data from the closed user group were used to generate data for analyzing the social structure revealed by the network traffic. RESULTS: The majority of communication within the closed user group was personal and not for professional purposes. The members of the CUG felt that the group improved their efficiency at work. CONCLUSIONS: The methods used present an interesting way to investigate the social structure surrounding communication via mobile phones. In addition, the benefits identified from the exploration of this closed user group make a case for supporting mobile phone closed user groups amongst professional groups.

Informatics lessons from using a novel immunization information system.

In the 1990s, NewYork-Presbyterian Hospital began developing a comprehensive, standards-based immunization information system. The system, known as EzVac, has been operational since 1998 and now includes information on 2 million immunizations administered to more than 260,000 individuals. The system exchanges data with multiple electronic health records, a public health immunization registry, and a standalone personal health record. EzVac modules have recently been incorporated into the OpenMRS application and are being used to enhance immunization efforts in developing nations. We report on our experience with developing and using the EzVac system for 1) clinical care, both in local and global settings, 2) public health reporting, 3) consumer engagement, and 4) clinical and informatics research. We emphasize the advantages of using health IT standards to facilitate immunization information exchange in each of these domains.

Using social networking to understand social networks: analysis of a mobile phone closed user group used by a Ghanaian health team.

BACKGROUND: The network structure of an organization influences how well or poorly an organization communicates and manages its resources. In the Millennium Villages Project site in Bonsaaso, Ghana, a mobile phone closed user group has been introduced for use by the Bonsaaso Millennium Villages Project Health Team and other key individuals. No assessment on the benefits or barriers of the use of the closed user group had been carried out. OBJECTIVE: The purpose of this research was to make the case for the use of social network analysis methods to be applied in health systems research--specifically related to mobile health. METHODS: This study used mobile phone voice records of, conducted interviews with, and reviewed call journals kept by a mobile phone closed user group consisting of the Bonsaaso Millennium Villages Project Health Team. Social network analysis methodology complemented by a qualitative component was used. Monthly voice data of the closed user group from Airtel Bharti Ghana were analyzed using UCINET and visual depictions of the network were created using NetDraw. Interviews and call journals kept by informants were analyzed using NVivo. RESULTS: The methodology was successful in helping identify effective organizational structure. Members of the Health Management Team were the more central players in the network, rather than the Community Health Nurses (who might have been expected to be central). CONCLUSIONS: Social network analysis methodology can be used to determine the most productive structure for an organization or team, identify gaps in communication, identify key actors with greatest influence, and more. In conclusion, this methodology can be a useful analytical tool, especially in the context of mobile health, health services, and operational and managerial research.

The importance of using open source technologies and common standards for interoperability within eHealth: perspectives from the Millennium Villages Project.

PURPOSE: The purpose of this chapter is to illustrate the importance of using open source technologies and common standards for interoperability when implementing eHealth systems, and to illustrate this through case studies, where possible. DESIGN/METHODOLOGY/APPROACH: The sources used to inform this chapter draw from the implementation and evaluation of the eHealth Program in the context of the Millennium Villages Project (MVP). FINDINGS: As the eHealth Team was tasked to deploy an eHealth architecture, the Millennium Villages Global-Network (MVG-Net), across all 14 of the MVP sites in sub-Saharan Africa, the team not only recognized the need for standards and uniformity but also realized that context would be an important factor. Therefore, the team decided to utilize open source solutions. PRACTICAL IMPLICATIONS: The MVP implementation of MVG-Net provides a model for those looking to implement informatics solutions across disciplines and countries. Furthermore, there are valuable lessons learned that the eHealth community can benefit from. ORIGINALITY/VALUE: By sharing lessons learned and developing an accessible, open source eHealth platform, we believe that we can more efficiently and rapidly achieve the health-related and collaborative Millennium Development Goals.

Capitalizing on the characteristics of mHealth to evaluate its impact.

The field of mHealth has made significant advances in a short period of time, demanding a more thorough and scientific approach to understanding and evaluating its progress. A recent review of mHealth literature identified two primary research needs in order for mHealth to strengthen health systems and promote healthy behaviors, namely health outcomes and cost-benefits (Mechael et al., 2010 ). In direct response to the gaps identified in mHealth research, the aim of this paper is to present the study design and highlight key observations and next steps from an evaluation of the mHealth activities within the electronic health (eHealth) architecture implemented by the Millennium Villages Project (MVP) by leveraging data generated through mobile technology itself alongside complementary qualitative research and costing assessments. The study, funded by the International Development and Research Centre (IDRC) as part of the Open Architecture Standards and Information Systems research project (OASIS II) (Sinha, 2009 ), is being implemented on data generated by 14 MVP sites in 10 Sub-Saharan African countries including more in-depth research in Ghana, Rwanda, Tanzania, and Uganda. Specific components of the study include rigorous quantitative case-control analyses and other epidemiological approaches (such as survival analysis) supplemented by in-depth qualitative interviews spread out over 18 months, as well as a costing study to assess the impact of mHealth on health outcomes, service delivery, and efficiency.

Combining vital events registration, verbal autopsy and electronic medical records in rural Ghana for improved health services delivery.

This paper describes the process of implementing a low-cost 'real-time' vital registration and verbal autopsy system integrated within an electronic medical record within the Millennium Village cluster in rural Ghana. Using MGV-Net, an open source health information architecture built around the OpenMRS platform, a total of 2378 births were registered between January 2007 and June 2009. The percentage of births registered in the health facility under supervision of a skilled attendant increased substantially over the course of the project from median of 35% in 2007 to 64% in 2008 and 85% midway through 2009. Building additional clinics to reduce distance to facility and using the CHEWs to refer women for delivery in the clinics are possible explanations for the success in the vital registration. The integration of vital registration and verbal autopsies with the MGV-Net information system makes it possible for rapid assessment of effectiveness and provides important feedback to local providers and the Millennium Villages Project.